The present invention is directed toward the domain of data processing, in particular toward the creation of qualitative, yet semantically meaningful distinctions at the earliest stages of processing of spatiotemporal data sets.
When confronted with spatiotemporal data, an intelligent system that processes the data to attempt to make sense of the ensuing stream may be overwhelmed by its sheer quantity. Video and other temporal sequences of images are notorious for the vast amount of raw data that they comprise. If, instead of two-dimensional images, data sets having three or more dimensions in addition to a temporal dimension, such as 3D magnetic resonance images, are processed, even a relatively small time sequence of data may overwhelm even the most powerful computers
One method for preventing the processing system from becoming overwhelmed may be to preprocess the data to indicate a measure of salience for different parts of the data with respect to the processing that is to be performed. Previous efforts that have attempted to abstract qualitative descriptors of motion information are relevant as identifying salient regions to be considered in motion processing. Much of this work is motivated by observations suggesting the inherent difficulty of dealing with the visual motion field in a quantitative fashion as well as the general efficacy of using motion in a qualitative fashion to solve useful tasks (e.g., boundary and collision detection). It should be noted, however, that the focus of most of this work is the qualitative interpretation of visual motion or optical flow. In this regard, the salience measure is generated using some of the motion processing steps.
An article by R. Nelson et al. entitled xe2x80x9cQualitative Recognition of Motion Using Temporal Texture,xe2x80x9d CVGIP-IU vol. 56, No. 1, pp 79-89 (1992) describes a method that treats motion information using temporal texture analysis. This method attempts to map spatiotemporal data to primitive, yet meaningful patterns. The analysis performed is based on statistics (e.g., means and variances) defined over normal flow recovered form image sequence intensity data. Furthermore, the patterns that it abstracts to (e.g., flowing water, flutter leaves) are specific and narrowly defined.
Considerable research has been concerned with effecting the recovery of image motion (e.g., optical flow) on the basis of filters that are tuned for local spatiotemporal orientation. Filter implementations that have been employed to recover estimates of spatiotemporal orientation include angularly tuned Gabor, lognormal and derivative of Gaussian filters. Also of relevance is the notion of opponency between filters that are tuned for different directions of motion, as disclosed in an article by R. Wildes xe2x80x9cA Measure of Motion Salience for Surveillance Applicationsxe2x80x9d Proceedings of the IEEE Conference on Image Processing, pp. 183-187 (1998). An essential motivation for taking such an operation into account is the close correspondence between the difference in the response of filters tuned to opposite directions of motion (e.g., leftward vs. rightward) and optical flow along the same dimension (e.g., horizontal).
Previous work also has been concerned with various ways of characterizing local estimates of spatiotemporal orientation. One prominent set of results along these lines has to do with an eigenvalue analysis of the local orientation tensor as disclosed in an text by G. Granlund et al entitled Signal Processing for Computer Vision, Kluwer Academic Publishers (1995). One goal of this analysis is to characterize the dimensionality of the local orientation as being isotropic, line or plane-like in order to characterize the local spatial structure with respect to motion analysis (e.g., distributed vs. oriented spatial structure with uniform motion).
The present invention is embodied in a method for generating a plurality of relevant spatiotemporal descriptors for a time sequence of multi-dimensional data sets. The method filters a volume of scalar quantities of the sequence of data sets corresponding to the time dimension and at least one dimension using a plurality of oriented filter characteristics. The filtered volume produces a plurality of spatiotemporal filter values corresponding to the scalar quantities of the data sets. The method determines the plurality of relevant spatiotemporal descriptors for the sequence of multi-dimensional data sets from the plurality of spatiotemporal filter values.